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ABSTRACT 



The purpose of this needs assessment, conducted as a class 
project, was to identify the gaps which may exist in the current 
Instructional Technology (IT) program at Utah State University (USU) . Five 
task groups were formed to handle specific areas of concern: (1) literature 

review, examining definitions of the field, competencies for graduates and 
professionals of the field, other programs in instructional technology, and 
exit interviews from USU graduates; (2) curriculum audit and curriculum 
matrix; (3) mail and e-mail survey of recent master's graduates of the USU IT 
Department; (4) telephone survey of other IT departments, in order to provide 
an external frame of reference; and (5) a focus group survey of companies in 
the surrounding area who hire instructional designers. Overall, the needs 
assessment was largely positive. In particular, graduates responded that the 
department teaches both the theoretical and practical aspects of the 
instructional design process well. Individual task group results are 
summarized, and the original reports provided by the task force teams are 
included. (AEF) 



***************************************************************************** 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



ED 434 649 



Utah State University 

Department of instructional Technology 

Needs Assessment 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 

J.N. Eastmond 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



r 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 
EDUCATIONAL RESOURCES INFORMATION 
/ CENTER (ERIC) 

Mfl This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Spring Semester 1999 



Submitted by 

Stephan Cartlidge 

Brent Gerity 

Dr. Nick Eastmond 

Instructional Technology 7010 

29 April 1999 



O 




FOREWORD 



The Utah State University Department of Instructional Technology is pleased to submit this 
Needs Assessment for departmental review. The Needs Assessment (NA) was conducted by 
Dr. Nick Eastmondis Spring 1999 Semester Instructional Technology (IT) 7010: Pro-Seminar II 
class. 

The purpose of the NA is to identify the gaps which may exist in the current Instructional 
Technology program. Additionally, the NA seeks to identify areas of redundancy where 
instruction overlaps beyond the point of efficiency. 

For purposes of our NA, we used the Seels and Richey 1994 definition of IT: 

Instructional Technology is the theory and practice of design, development, utilization, 
management, and evaluation of processes and resources for learning. 

Additionally, we used the Witkin and Altschuld 1995 definition of NA as: 

A systematic set of procedures undertaken for the purpose of setting priorities and making 
decisions about program or organizational improvement and allocation of resources. 

The priorities are based on identified needs. 

A summary videotape is nearing completion to report the results of this study. 
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Overall Results of the Needs Assessment 



Methods 

Our approach to the NA as a class was to divide into the following five task groups to 
handle specific areas of concern: Literature Review, Curriculum Audit and Curriculum 
Matrix, Mail and E-mail Survey, Telephone Survey, and Focus Groups. Each task group 
targeted a specific audience and used the Seels and Richey definition of IT as a basis for 
data collection. This section summarizes the particular methods of each task group. 

Literature Review. 

The literature review analyzed three aspects of the field of Instructional Technology and 
evaluated the Department of Instructional Technology at Utah State University. This 
review included: 

1 . definitions of the field 

2. competencies for graduates and professionals of the field 

3. other programs in instructional technology 

4. exit interviews from USU graduates 

One of the main purposes of this NA was to revise the curriculum; therefore, we 
researched background on the previous three categories. Part of every subject curriculum 
is historical background and theory. The IT field is constantly evolving and the 
definitions of the field have changed to reflect this evolution. The Literature Review 
examined several other programs considered “leaders” in IT to compare their curriculum 
with that of USU. The final category provides an internal examination of our own 
program. 

Curriculum Audit and Curriculum Matrix: 

The Curriculum Audit uses a combination of interviews and document reviews to collect 
information for the curriculum audit worksheets and the curriculum matrix. Dr. Smellie 
was selected to be interviewed because of his position as department chair. Dr. Wolcott 
was selected due to her current position as a faculty member and her future position of 
interim department chair for the 1999-2000 school year. 

The Curriculum Audit is an instrument adapted from materials written by M. D. Thomas 
and J. H. Brewer, "Educational Auditing: A Guide to School Effectiveness." The audit 
materials are currently unpublished, but the copyright is held by the authors and 
Associated Consultants in Education. The Curriculum Audit is designed to determine if 
the conditions of effective teaching and learning are present in a secondary school setting. 
The audit was adapted for use in this needs assessment by substituting the word 
department" for "school," and "college" or "university" for "school district." 

Mail and E-mail Survey: 

The purpose of the Mail and E-mail survey was to contact recent Master's graduates of 
the USU IT Department to assess the relevance of the Instructional Design (ID) skills 
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they gained from their studies with respect to their current job. We generated a contact 
list of IT Department Master's graduates since 1994. Secondly, we mailed approximately 
110 letters and surveys asking respondents to reply via mail or through an on-line survey. 
We sent an e-mail letter was to an additional 20 respondents requesting a response 
through the on-line survey, creating a total sample size of 130. 

Telephone Survey: 

The Telephone survey was conducted in order to provide an external frame of reference 
for the departmental NA because findings from internal reviews carry considerable 
validity. Input from other IT departments also allows us to compare and consider our 
own conclusions within a broader context and to note any major divergences from 
important national trends. We chose most of the larger instructional programs in the 
country and a few of the smaller programs. The majority of telephone interviews were 
conducted with departmental chairs. Two interviewers conducted the surveys by using 
an identical set of 20 open-ended questions. The initial question in any given area often 
led to more refined questions. The overall tone of the Telephone survey was 
conversational and the average interview length was half an hour. 

Focus Group: 

The Focus Group survey was conducted to 1) gather information about which companies 
hire instructional designers from the USU IT department, and 2) to determine what skills 
and experience those companies look for in potential employees. The Focus Group 
began by identifying companies in the Provo, Ogden, and Salt Lake City area who hire 
Instructional Designers. We contacted the personnel departments of those companies and 
asked a representative to meet with us to discuss their experiences in hiring IT graduates. 
The meetings took approximately one and a half hours, and were conducted informally. 
We met with representatives from the following six companies: TenFold Corporation, 
Novell, American Stores, Allen Communications, Inc., Utah Transit Authority, and the 
CES Department of the Church of Jesus Christ of Latter Day Saints. Following the 
meetings, we conducted a follow-up phone call to validate the information gathered 
during the focus group: this technique is called “member checking” and helps ensure the 
quality of qualitatively gathered information. 

Limitations of the Study 

In general, the NA study was limited by time constraints and the accuracy of information 
provided by participants. Also, not all of the task groups were able to receive as much 
information as they would have liked: the Mail and E-mail survey only yielded a 32% 
return rate. While this is below the conventional 40% benchmark for statistical 
reliability, the results are reported here for completeness and to provide a minimal level 
of face validity. Additionally, these results were meant to overlap with other task group 
findings in order to build a rich body of data. 

The USU IT Department’s Ph.D. program recently moved from being an interdisciplinary 
to a departmental degree and has changed significantly. Additionally, the 1998-1999 
school year saw an institutional transition from the quarter to semester system; the IT 
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Department curriculum also shifted, and this NA does not take the new curriculum into 
account. 

Results 

Overall, this NA was largely positive and reflected well upon the USU IT Department. 
In particular, graduates responded that the department teaches both the theoretical and 
practical aspects of the ISD process well. Additional positive findings are detailed in the 
individual task group reports. 

The NA revealed gaps in the following areas: 

1. There needs to be more fidelity between the classroom environment and the context 
of the workplace. Most of the USU IT students will work in the corporate 
environment upon graduation, and they will require a sense of operating as a 
professional in the business world. 

2. There needs to be even more integration of the applied skills and tools classes with 
the learning theory and instructional design classes. Applied skill and tool classes are 
important but must be taught in the context of instructional design, with real 
problems. 

3. There needs to be more emphasis on technical and professional writing, and more 
opportunities for students to improves their writing skills. Many graduates reported 
inadequate instructor feedback on their writing. 

4. There needs to be more emphasis on emerging Internet technologies and web-based 
skills. The USU IT Dept, is not competitive with other similar programs in this arena. 

5. There needs to be more opportunity for students to have first-hand teaching 
experience. TA-ships, presentations, and occasional opportunities to lead classroom 
discussion do not build adequate corporate leadership skills. 

The remainder of this section details the result-specific findings of each task group. 

Literature Review Results 

The Literature Review data does not provide insight into departmental needs per se, but 
was a valuable point of reference for this NA. 

Curriculum Audit and Curriculum Matrix Results 

The Curriculum Audit indicates that improvement is needed in the following areas: 

• Connecting the budget with curriculum planning and development efforts 

• Coordinating the written, taught, and tested curriculum across classes and programs 

• Creating assessments to measure effectiveness of instruction and monitor 
coordination between classes 

The curriculum matrix shows that there are curricular gaps in the following areas: 

Masters Level 

• Management, utilization, and evaluation theory and practice 

• Ethics issues receive brief attention in InsT 6080: Instructional Technology Core 
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• Dr. Smell ie expressed concern that there isn't enough development practice 
Ed.S. andPh.D. Levels 

• Management and utilization theory and practice 

• Design and development practice 

• Ethical issues addressed briefly in classes, such as InsT 6080 

Mail and E-mail Survey Results 

The Mail and E-mail survey data reveals one obvious strength and two obvious 
weaknesses in the program. Respondents rated presentation skills at above average 
importance and they rated USU's preparation in this skill equal to or greater than their 
perceived need. Technical writing skills were rated at above average importance but 
USU's preparation in this skill was far less than their perceived need. Web authoring 
skills were also rated at above average importance but USU's preparation in this skill was 
significantly less than their perceived need. 

From the data, it is apparent that front-end analysis and design is considered of high 
importance to recent master's graduates with USU preparation in this are nearly equal to 
the perceived need but falling slightly short in each category. The smaller variance of the 
responses related to front-end analysis gives weight to the consistency of the perceptions 
amongst respondents. 

All remaining skills and topics except two showed the respondent's perception of 
importance to their current job exceeding their perception of the adequacy of their 
training in that skill at USU. Only in CBT authoring and digital resource creation did 
their perception of USU preparation exceed their perception of its importance in their 
current job. 

The skill rated as the least important in their current job was CBT authoring. However 
this same rating had the highest variance demonstrating the broad range in responses 
from those surveyed. The skills rated the most important where front-end analysis and 
presentation skills. 

Telephone Survey Data Results 

The Telephone Survey data shows that USU should: 

• Integrate tool courses and instructional design courses, over a multi-course sequence 
if necessary. Continue to require tool courses, but only teach them within an 
instructional design framework. 

• Define appropriate organizational change or management theory courses for 
instructional technology students and build them into the required curriculum. Work 
with other departments if the department itself is unable to offer them without 
support. The overwhelming majority of IT students will be working in a corporate 
environment. A complete view of IT includes the context in which it is practiced. 

• Determine processes and procedures through which an instructional design 
perspective can be brought to bear on courseware development for the Web on a 
University-wide basis. 
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• Find ways to involve recent graduates working nearby in certain courses. Let them 
teach or co-teach a course, under faculty supervision, that is designed to work with 
other elements referred to in the second point above. 

Focus Group Data Results 

The Focus Group data provided the following main points: 

Specific Skills and Experiences Necessary to Instructional Designers — 

• ISD Process 

• Teamwork 

• Communication/Writing 

• Technical Knowledge 

• Critical Thinking 

• Affective Attitudes 

• Customer-oriented approach (able to communicate with customers) 

Weaknesses in Recent Graduates — 

• Lack of knowledge about the business world 

• Lack of teaching skills and experience 

Future Needs and Recommendations 

Due to this year’s shift in curriculum, one of our primary recommendations is a follow-up 
study which addresses recent changes. Secondly, future NAs must begin with more 
specific questions (i.e., asking graduates “what got you your job?”) in order to delimit 
results and provide feedback which can be acted upon. Third, Ph D. and Ed.S. students 
should be included in future quantitative and qualitative data collection. Finally, efforts 
should be made to target a large sample size in all data collection methods; this problem 
may be inherent in the NAs time constraints. 
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Individual Task Group Results 

The following sections contain the original reports provided by individual Needs 
Assessment Task Force teams. 
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NEEDS ASSESSMENT 
LITERATURE REVIEW 



I. Scope 

This literature review took a national view of three aspects of the field of instructional technology and 
also took a look at the Department of Instructional Technology at Utah State University: 

1 . definitions of the field 

2. competencies for graduates and professionals of the field 

3. other programs in instructional technology 

4. exit interviews from USU graduates/curriculum audit 

As one of the main purposes of the needs assessment of the Department of Instructional Technology 
at Utah State University was to revise the curriculum, background on these three categories was 
researched. Part of every subject curriculum is historical background and theory. This field is 
constantly evolving and the definitions of the field have changed to reflect this. Competencies 
required from graduates to be successful professionals in their field will be a necessary item used to 
revise the curriculum. Finally, other programs considered “leaders” in instructional technology were 
examined to compare their curriculum with Utah State University. The last category, “exit 
interviews from USU graduates/curriculum audit” gives us an internal look at our own program. 
Daniel House and Barry Bratton wrote an article on the evaluation of curriculum trends in IT doctoral 
programs in 1989, and suggested that “instructional technology graduate programs should not delay 
in increasing course offerings in instructional development and the use of new technologies. .. 
Instructional technology programs are incorporating curriculum topics in areas with increasing 
employment opportunities for graduates. . .” (House, 1989). 

n. Limitations 

Even though these items were discussed in class, when starting this literature review, it was seen that 
more complete and specific questions were needed to refine the search; also, because of time 
constraints, this review might have been a little more thorough. Instead of an all-encompassing 
review of the definitions of the field and graduate competencies, most of the review covered only the 
most current information. Because of the desire to perform curriculum revisions in the Department 
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of Instructional Technology, information prior to the early 1990’s was considered outdated, although 
there were many useful and relevant studies done in this time period. Besides current articles and 
monographs, one particularly useful place to find information of this nature is in personal or telephone 
interviews from leaders, professionals, and faculty members in the field. This was done as part of the 
needs assessment through the use of telephone interviews to faculty members from other instructional 
technology departments around the country and focus groups of recent graduates and professionals 
in the field Reports were made separately on these findings. This comprises the last category of 
“other instructional technology programs”. 

III. Findings 

Definitions of_ the Field 

ADefinitions do not create a field but, rather, help to explain its purposes, functions and roles to those 
within and those outside the field. 0 (Reiser and Ely, 1997) As the field is constantly evolving, 
definitions and terms also change. In this review, the term Ainstructional technology© will be used 
as the broad term to define the field; another very commonly used term is Aeducational technology© . 
The first formal definition of the field was approved and published by the Commission on Definition 
and Terminology, which was established by the Department of Audiovisual Instruction (DAVI) of 
the National Education Association. It was entitled The Changing Role of the Audiovisual Process 
in Education: A Definition and a Glossary of Related Terms (Ely, 1963). This definition stated: 

Audiovisual communication is that branch of 
educational theory and practice concerned 
primarily with the design and use of mess- 
ages which control the learning process. 

In 1972, the DAVI Commission on Definition and Terminology produced a new definition of the 
field, changing the field name from audiovisual communications to educational technology: 

Educational technology is a field involved in 
the facilitation of human learning through 
the systematic identification, development, 
organization, and utilization of a full range 
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of learning resources and through the manage- 
ment of these processes. 



The longest standing definition came from the Association for Educational Communications and 
Technology in 1977 and consisted of sixteen parts in sixteen pages. The first line of the definition 
reads: 

Educational technology is a complex, 
integrated process involving people, 
procedures, ideas, devices, and organ- 
ization, for analyzing problems and 
devising, implementing, evaluating, 
and managing solutions to those problems, 
involve in all aspects of human learning. 

The most recent definition of the field came about in 1994 and was published by the AECT as 
Instructional Technology as The Definitions and Domains of the Field (Seels & Richey, 1994). 

Instructional Technology is the theory and 
practice of design, development, utilization, 
management, and evaluation of processes 
and resources for learning. 

This definition describes the five domains of the field, which include design, development, u tilizat ion, 
management, and evaluation. 

According to Reiser and Ely, the major changes in these definitions throughout the years fall into 5 
main categories: 

1 . the focus of definitions (and the field) 

2. the functions performed by professionals 

3 . the products they work with 

4. the role those products play in the instructional environment 

5. the goal of professional efforts 

The authors point out that the shift in the focus of the definitions went from media to messages 
(1963), to systematic design process (1970s), to the current de fin ition focusing on the five domains, 
which are all related to processes and resources for learning. All of these changes reflect changes in 
the field . While professionals used to focus on the utilization of media, the functions broadened to 



include design, then evaluation, development, management, analysis and organization (Reiser & Ely, 
1997). 

New ideas and technologies will affect the way we define the field of Instructional Technology in the 
future. Ideas that have already had an impact on the field include distance learning, the 
constructionist movement, cooperative and collaborative learning, performance technology, the use 
of electronic support systems, and the use of networks for instructional purposes (Reiser & Ely, 
1997). 

Donald Ely put together a short digest of frequently asked questions about the field of instructional 
technology in 1993, and an updated version in 1997. These digests provided some background 
information and sources to help understand the concept of educational technology. Some of the 
questions that were posed included: 

1. Where do educational technologists obtain professional education? 

Professional programs are offered mostly at the graduate level, although there are a few two-year 
postsecondary programs in junior and community colleges. 

2. What do instructional technologists do? 

Most carry out one or a few functions performed in the field, including instructional design, 
production of instructional materials, or managing instructional computing services or learning 
resources collections, to name a few. 

3. Where are they employed? 

Most are employed in educational settings, such as schools and colleges as directors or resource 
learning centers and developers of curriculum materials, or colleges and universities as instructors or 
those involved in instructional improvement programs. More recently, the trend has been to employ 
instructional technologists in business, industry, government, military, and in the health professions 
(Ely, 1997). 



There weren’t many changes in the prominent resources or definitions of the field between the 1993 
and the 1997 versions of the digest. 

Competencies of_ IT Graduates 

Competencies of instructional technology graduates differ somewhat depending on whether they are 
masters degree or doctoral graduates, and in what type of institution they are employed (business & 
industry, educational, govemment/military). The main differences in competencies required from job 
applications were that masters program graduates were expected to be proficient in project 
management and instructional design and development, and doctoral graduates were required to have 
experience in teaching, conducting research and writing grant proposals. Both groups, however, 
were expected to have experience and knowledge in computer technology and multimedia production 
(Moallem, 1995). 

Several studies on the competencies of IT graduates were examined, but the study most relevant to 
the needs assessment at USU was a study done in 1995 that analyzed job announcements in the field 
of instructional technology to illustrate required skills and competencies for IT graduates from three 
different institutions: business & industry, government and military, and university/college/school 
district (Moallem, 1995). According to this study, the top five skills/knowledge rated by all three 
institutions were similar, and included teaching/experience, knowledge/experience in instructional 
design & development, knowledge/experience in advanced media/technology, knowledge/experience 
in instructional computing (CBI, CAI), knowledge/experience in project management, 
knowledge/experience in computers (hardware, software, authoring system), knowledge/experience 
in curriculum and material development, and knowledge/experience in evaluation. Excellent 
summary charts were given. 

A similar study was published in 1993 to determine whether professionals in business, health, 
agencies, and military were receiving the training necessary to work as instructional designers and 
trainers. The survey used was based on competencies perceived as essential by a group of university 
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professors. The highest ranked competencies (89.4% to 100% of the respondents considered these 
competencies to be important of high priority) included (Morlan, 1993): 

$ knowledge, understanding and application of instructional design models and principle 
$ learning needs assessment and evaluation/understanding skills, and applications 
$ project management, from inception to completion 
$ design, production and utilization of self-paced learning materials 
$ instructor-led training, including skills necessary for giving effective presentation 
$ design, production and utilization of independent learning modules 
The majority of the respondents worked in business and industry or educational settings; most had 
either a bachelors or masters degree; and most degrees were from the instructional technology, 
business, educational and social studies fields. 

A third study, published in 1996, had a slightly different focus. This study focused on Aproviding 
information on the implementation of instructional technology in employee training and the 
competencies needed by trainers to utilize instructional technology in their jobs© (Furst-Bowe, 
1996). Respondents to the survey described how instructional technology was being used to design 
and deliver training in their organizations, their ideas about the types of technology that might be used 
to design and deliver training in the future, the level of competency needed by trainers to utilize each 
type of technology, and sources of competency development for each technology. One section of the 
survey also asked them to identify Abarriers in the workplace which limited the implementation on 
instructional technology in training© (Furst-Bowe, 1996). The survey indicated that twelve 
technologies were being used by at least 50% of the respondents, and these included computer-based 
training, computer tutorials, computer simulations, computer presentation systems, presentation 
software, electronic performance support systems, on-line help systems, information databases, 
multimedia systems, LCD panels, LCD video/data projectors, and local area networks (Furst-Bowe, 
1 996). These same technologies were perceived to be important for the next three years. The most 
frequently needed competency reported by the respondents was the ability to use or assist trainees 
in the use of technology; two others included the ability to evaluate a specific technology=s 
effectiveness and the ability to develop programs or systems for most technologies. The most 
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common barrier to implementing instructional technology in the workplace was lack of time and 
financial resources (cited by over 75% of respondents). Second most cited reasons included lack of 
compatibility between systems, lack of management support and technical support, and lack of trainer 
skills. Interestingly, in this study, the most common source for competency development in 
instructional technology was self-study and vendor-sponsored training, followed by seminars, 
conferences, and training programs sponsored by professional organizations. Colleges and 
universities were found to play a very minimal role in providing trainers with skills in computer-based 
training, multimedia systems, EPSS, distance earning systems, or computer presentation systems 
(Furst-Bowe, 1996). 

The final study, published in 1988, provided information on the relationship between academic 
preparation in IT programs, competency attainment, student characteristics, and job success. This 
study was considered too out-of-date for review purposes. 



Programs in Instructional Technology 

Other programs were examined through telephone interviews with faculty from programs around the 
country. The programs that were picked are recognized as leaders in the field, with a few others 
thrown in for good measure. All programs offered masters and doctoral degrees in instructional 
technology. 



IV. Future Considerations 

In the future, a more comprehensive review or meta-analysis might be done, but due to time 
constraints, a more brief look at the field was taken at this time. More specific questions should be 
posed at the start to reflect the purposes of the assessment and to narrow the search. For example, 
instead of “finding information on the field, competencies, and trends”, which was the broad basis for 
research at the start of this review, we might ask ourselves questions like: 1) What courses do other 
programs, especially the more distinguished programs in the field, contain in their curriculum? What 
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are their objectives? What types of things do they emphasize in their programs? 2) How well do 
recent graduates feel they have been prepared to enter the job market? Do they see the need for any 
particular training they may or may not have received? 3) What competencies do employers look for 
when hiring a new instructional designer? 4) What are the differences in the competencies required 
of different institutions, specifically business, industry, govemment/military, and educational 
positions? Can the curriculum be altered to cater to these needs? 5) What are the recent trends in 
graduate IT programs? What are the recent trends in the workplace? 6) What is the program at 
USU currently lacking? What are we doing in excess? 

While all of these (and others as well) are questions that the needs assessment seeks to answer, it 
helps to have these questions in mind beforehand when doing the literature review. It is possible that 
not all of these questions may be answered with recent literature, and these areas may be items for 
further study or publication. 

When examining other programs, a larger random sample might be taken, and analysis should be more 
in depth. A list of websites of instructional technology programs can be found at the following 
address: http://www.intranet.csupomona.edu/-urviers/ist/ist .html With more time, web sites of all 
of these programs could be analyzed for their objectives and curriculum to give a more complete 
picture of the programs. On the other hand, relying solely on web sites to obtain information on a 
program may not be completely objective. Sites may be out of date or incomplete. Telephone 
interviews are a good source for general information, but a real in-depth analysis may be more helpful. 



Only one article, published in 1989 was found on trends in IT doctoral programs. This, as well as 
more current literature on competencies of graduates and professionals may be an avenue for future 
studies or publications. -As the field keeps changing, new or revised definitions for the field might 
be considered as well. 
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Instructional Technology Department 
Curriculum Audit and Curriculum Matrix 
Prepared by Mary Ann Pariin and Daren Olson 
April 20, 1999 

I) Sampling methods 

The Curriculum Audit 

This audit used a combination of interviews and document reviews to collect information for the 
curriculum audit worksheets and the curriculum matrix (See appendix A and B). Dr. Smellie was selected 
to be interviewed because of his position as department chair. Dr. Wolcott was selected due to her current 
position as a faculty member and her future position of interim department chair for the 1999-2000 school 
year. Unless indicated otherwise, the reporting of any statements in this document should be considered to 
be paraphrases and not actual quotes from those interviewed 

The curriculum audit is an instrument adapted from materials written by M. D. Thomas and J. H. 
Brewer, "Educational auditing: A guide to school effectiveness." The audit materials are currently 
unpublished but the copyright is held by the authors and Associated Consultants in Education The 
curriculum audit is designed to determine if the conditions of effective teaching and learning are present in 
a secondary school setting. The audit was adapted for use in this needs assessment by substituting the word 
"department" for "school," and "college" or "university" for "school district" The curriculum audit is based 
on the following assumptions: 

"There is no way a large, complex organization can direct its energies over time to accomplish specific 
results without leaving behind a paper trail that shows the linkages between what it wanted to 
accomplish (objectives) and what it did accomplish and how organizational action (behavior) was 
altered to attain the objectives (titles of documents notwithstanding) . A system which is consistently 
improving student achievement will leave behind a trail of documents which will show how they 
identified objectives and how they were translated intact from the policy/strategic levels to the 
operational levels. The curriculum audit uses multiple methods of data gathering in order to verify 
findings in the audit. In order to appear in the audit a feet must be verified from at least two (hopefully 
more than two) sources. These sources may include, but are not limited to: 

a) Documents: policies, memoranda, contracts, guides, linkage documents, manuals 

b) Interviews: key participants in the design and delivery of the curriculum focusing on 
interrelationship among documents and implementation 

c) Site visits: observation of the context for curriculum delivery, noting potential discrepancies and 
other factors affecting delivery." 

The curriculum audit is also made against criteria in the following areas: 

1 . Control: The department is able to demonstrate its control of resources, programs, and personnel. 

2. Direction: The department has established clear and valid objectives for students. 

3 . Connectivity and Equity: The department has documentation explaining how its programs have been 
developed, implemented, and conducted 

4. Feedback: The department looks for results from department designed or adopted assessments to 
adjust, improve, or terminate ineffective practices. 

5. Productivity: The department has been able to improve productivity. 

The assumptions and criteria of the curriculum audit clearly show that it is an objective-based 
approach. In addition, it asks for strict assessments to assure that instruction leads to successful completion 
of the objectives. Since effectiveness and productivity is measured through objectives, and a curriculum- 
driven budget depends upon these assessment measures in order to justify expenditures. If the department's, 
or even individual professors, do not share in this objective-based philosophy, it may be inappropriate to 
base any budgetary decisions or make curricular changes based on this curriculum audit However, if the 
department finds that it shares this philosophy, the relevance of these findings increases significantly. It is 
hoped that no matter what the philosophy espoused by the department or its members that this curriculum 



audit may be a source of reflection and maybe spur further investigation into how the curriculum may be 
improved. 

The Curriculum Matrix 

The curriculum matrix was created to see how well the written course descriptions matched the 
definition and domains of the field as outlined by Seels and Richey (1994) in their book, "Instructional 
technology: The definition and domains of the field." Sources for this information include masters, 
educational specialist, and doctoral program guidelines published by the department and the Utah State 
University 1999-2000 General Catalog. 

Interview data was collected in the form of notes and worksheets. Current program documents and an 
older strategic plan were also collected from the department. Not all professors were interviewed for the 
curriculum audit. This was due in part to time constraints. In addition, we determined that new faculty 
members, part time faculty members, and absent faculty members should not be included as sources of 
information. Since one faculty member, Dr. Eastmond, was in charge of conducting the needs assessment, 
it was felt that he should not be included as well. Because it is a small deportment, that only left three other 
teachers as potential sources of information in our sample. Since the audit does not rely on the collection 
of opinion, it was determined that it would be sufficient to only interview Dr. Smellie and Dr. Wolcott. 

II) Limitations of the Curriculum Audit and Curriculum Matrix 

The curriculum audit was created for use in a secondary school setting. Therefore, some of the 
questions were not applicable to a university department setting Dr. Wolcott stated that she felt the 
curriculum audit was not completely relevant to higher education and that it didn't ask the right questions. 
This was based on the belief that the department is more geared towards being responsive to the market 
place, instead of responding to the typical bureaucracy associated with school districts. In addition, she 
believed that the emphasis of the department curriculum is preparation for specific jobs, and that the 
curriculum audit should be assessing whether or not the curriculum is pitched at the right level to insure job 
success and acceptance into the field This contrasts with secondary school curriculum in that it often 
prepares students for post-secondary education instead of for direct placement into the workplace. Finally, 
Dr. Smellie pointed out that many of the curriculum issues mentioned in the audit are dealt with at a 
university level and are covered by university policies. Therefore, it is not necessary to restate those 
policies again at the department level. In addition, both Dr. Smellie and Dr. Walcott stated that at the 
university level a certain amount of professionalism is expected by all within the department regarding 
some of these issues. Thus, certain working policies may be in place even though they are not explicitly 
stated or recorded on paper. 

The language of the curriculum audit is difficult to decipher in places. This may lead to 
misinterpretation of the meaning of certain questions. During the interviews, the meanings of these 
questions were negotiated between the interviewer and the subjects. This was done in an effort to extract as 
much information as possible, since for the general purposes of this audit the meaning of the question was 
secondary to the information that it uncovered. In short, some information from a misinterpreted question 
was felt to be better than no information at all. 

The curriculum matrix is also designed to give a broad overview of the curriculum. Its purpose is to 
place classes within the general categories suggested by Seels and Richey's definition of the field. It is not 
meant to cover everything that may be taught in a class. There are many opportunities for instruction within 
all of these domains in a single class. Therefore, certain domains of the field may be under or over- 
represented in the matrix. However, we believe that the categories chosen reflect an overall placement 
within the Seels and Richey framework and allow one to develop a general sense of the curriculum's 
breadth and depth. 

III) Results 
Curriculum Audit 

The curriculum audit indicates that improvement is needed in the following areas: 

• Connecting the budget with curriculum planning and development efforts. 

• Coordinating the written, taught, and tested curriculum across classes and programs. 

• Creating assessments to measure effectiveness of instruction and monitor coordination between 

classes. 
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Curriculum Matrix 



The curriculum matrix shows that there are curricular gaps in the following areas: 



Masters 

• Management, utilization, and evaluation theory and practice. 

• Ethics issues receive brief attention in InsT 6080: Instructional Technology Core 

• Dr. Smellie expressed concern that there isn't enough development practice. 

Ed.S. andPh.D. 

• Management and utilization theoiy and practice. 

• Design and development practice. 

• Ethical issues addressed briefly in classes, such as InsT 6080. 

TV) Conclusions 

• The masters program is heavily oriented towards design and development Dr. Smellie questions if 
there is time in the curriculum to provide instruction in all of the areas outlined by Seels and Richey. 

• The Ph.D. program lacks some of the design and development practice in the masters program. This is 
due in part to the admissions requirements of having a masters degree already before entering the 
Ph.D. program. It is assumed that Ph.D. students have some of this experience coming into the 
program, and that further instruction may be too repetitive. There are, however, eight credits of 
practicum classes designed to provide this experience. 

• Dr. Smellie believes that improvement is needed in the pacing of research classes at the Ph.D. level. 
There is a current concern that students are staying in the program too long. This is addressed 
somewhat through InsT 7810: Research Seminar and a yearly Ph.D. review process. 

V) Future Needs 

Because of the recent move from a quarter system to semesters, more time is needed to fine-time the 
curriculum. Future needs assessments should examine the steps that have been taken to adjust the 
curriculum. In addition, as the department undergoes staff and leadership changes, there should be further 
refinement of the curriculum as the expertise of new staff members is applied to those areas where 
improvement is required. 

VI) Raw Data Packet 

• Appendix A: Curriculum Audit Worksheet Results 

• Appendix B: Curriculum Matrix 

• Appendix C: Instructional Technology Department Strategic Plan 



Appendix A: Curriculum Audit Worksheet Summary 



The following worksheet summary shows the combined responses of Dr. Smellie and Dr. Walcott, along 
with any comments they made in response to the questions. Dr. Smellie's ratings and comments will be 
marked with a capital "S M and Dr. Walcott's with a capital "W." Any explanatory notes by the interviewer 
will be marked with a capital "1." If there were questions concerning the meaning or relevance of a 
particular question, the initials will be followed by a question mark "? H . 

It should be noted that these questions were originally designed to determine if written policies existed 
covering these areas. However, since there were not many written policies in the department, the worksheet 
responses were adapted to reflect opinions on whether or not these areas were being addressed adequately. 
Differences in responses are due in part to the different interpretations of the questions. In addition, both 
Dr. Smellie and Eh. Walcott sometimes emphasized different things when they considered these questions, 
therefore their responses may reflect a variety of concerns instead of the exact same concern. 



Components of a curriculum-driven budget 

I: This worksheet determines if the department budget is driven by 
curricular issues or by some other source. 


Adequate 


Partially 

Adequate 


Inadequate 


1 . Tangible, demonstrable connections are evident between 
assessments of operational curriculum effectiveness and 
allocations of resources. 


s 


W 




2. Priorities in budget process are set by participation of key 
educational staff in the decision-making process. 




S 


sw 


3 . Teacher and department head suggestions and ideas for budget 
priorities are incorporated into the decision-making process. 




s w 




4. Rank ordering of program components is provided to permit 
flexibility in budget expansion, reduction, or stabilization based on 
changing needs or priorities. 


s 




w 


5. Cost benefits of components in curriculum programming are 
delineated in budget decision-making. 




s 


w? 


6. Each budget request or submittal shall be described so as to 
permit evaluation of consequences of funding or non-funding in 
terms of performance or results. 




s 


w? 


7. Budget requests compete with each other for funding based 
upon evaluation of criticality of need and relationship to achievement 
of curriculum effectiveness. 


s 




w 



Comments: 

S: During his time as department chair he has never turned down requests for resources from professors. He 
has tried to be supportive of budgetary requests, even though he feels that the department needs at least an 
extra $10,000 a year in monies from the state. Most of the money in the budget goes to salaries and 
operating costs. Off-campus programs are self-funded. 

W: Things run smoothly. She believes they have necessary resources to have an effective curriculum. She 
also believes the operating budget is too small. 

I: Both seemed to think that although there is room for improvement, the department handles the budgetary 
decisions appropriately and that the major need for change was in the amount allocated to the department. 
Dr. Smellie mentioned that the new building was an improvement over the previous offices, classrooms, 
and computer labs. The program is also slated to get new computers for its labs on a set schedule. 
Therefore, most of the time it may seem like the computers are behind the market. 
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Characteristics of Good Policies on Curriculum Management 

There are written, directive statements of policy that cover the following criteria: 



Control 


Adequate 


Partially 

Adequate 


Inadequate 


A. An aligned written, taught, and tested curriculum 


w 


[ S 




B. Philosophical statements of curriculum approach 






sw 


C. Department adoption of the curriculum 
S: Curriculum is agreed upon through informal channels within the 
department. The university policy outlines curriculum requirements 
for new classes. 

W: Department adoption is loosely done. 


s 


W 




D. Accountability through roles and responsibilities 


sw 


s 




E. Long-range planning. 




s 


w 



Direction 


Adequate 


Partially 

Adequate 


Inadequate 


A. Written curriculum for all subject/1 earning areas 
S&W: This is covered bv the course syllabi. 


S 


W 




B. Periodic review of the curriculum 


S 


W 




C. Textbook/resource adoption by the department 
S&W: This is individually determined by the instructors. 


s 


W?(n/a) 




D. Content area emphasis 


s 


W 


w 1 



Connectivity and Equity 


Adequate 


Partially 

Adequate 


Inadequate 


A. Predictability of the written curriculum from one level to 
another 

W: N/A-There is good connection of programs. 




S 




B. Vertical articulation and horizontal coordination 
S.N/A 




W 


W 


C. Training for staff in the delivery of the curriculum 
W: N/A 




S 




D. Delivery of the curriculum 
S:N/A 

W: Implicit in scheduling. 








E. Monitoring of the delivery of the curriculum 

S: Use student evaluations to monitor classes, as required by 

university policy and the provost's office. 


s 


w 


w 


F. Equitable access to the curriculum 

S: Statements concerning equitable access and ADA requirements in 
department handouts 


sw 







Feedback 


Adequate 


Partially 

Adequate 


Inadequate 


A. An assessment program 
S: Conducted every five to six years. 


s 


S 




B. Use of data from assessment to determine program/curriculum 
effectiveness and efficiency 




SW 




C. Reports to the department about program effectiveness 
S: These are given in faculty meetings. 


sw 
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Productivity 


Adequate 


Partially 

Adequate 


Inadequate 


A Program-centered budget 

W: The department doesn't use this type of budgeting. 


s 


W 


W 


B. Resource allocation tied to curriculum priorities 


s 


w 




C. Environment to support curriculum delivery 


sw 






D. Data driven decisions for the purpose of increasing student 
learning 

S: This is up to the faculty. 






W 



I: According to Thomas and Brewer, "in order for the policies to be considered sufficiently adequate to 
ensure curricular quality, 70% or more of the criteria of good policies need to be present." Again, there 
seems to be agreement that some improvements could be made concerning these policies. However, the 
general opinion is that it is at least somewhat adequate. The only agreement about an "inadequate" policy 
centered on the philosophical statements of the curriculum approach. This should not be interpreted to 
mean that both thought there needed to be a single philosophical statement about the curriculum approach, 
but rather that there were no such statements, either implicit or explicit within the department This again is 
left up to the individual professors. 
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Standards Review 



Standard 1 

Control: The department is able to demonstrate its control of 
resources, programs, and personnel. Common indicators are: 


Adequate 


Partially 

Adequate 


Inadequate 


1) A curriculum that is centrally defined and adopted by the 
department 


sw 






2) A clear set of policies that establish an operational framework 
for management that permits accountability 


s 


W 




3) A clear set of policies that reflect state requirements and local 
university goals and the necessity to use achievement data to 
improve school system operations. 

S: N/A 

W: Need an assessment program first. 




W 




4) A functional administrative line of authority that facilitates the 
design and delivery of the department's'curriculum 


sw 






5) A direct, uninterrupted line of authority from university/college 
officials to department head and instructors 


sw 






6) Organizational development efforts which are focused to 
improve system effectiveness 


s 




w 


7) Documentation of the college/department planning for the 
attainment of goals, objectives, and mission over time 


s 


W 




8) A clear mechanism to define and direct change and innovation 
within the school system to permit maximization of its resources 
on priority goals, objectives, and mission. 


s 


W 





Standard 2 

Direction: The department has established clear and valid objectives 
for students. Common indicators are: 


Adequate 


Partially 

Adequate 


Inadequate 


1 ) A clearly established, system-wide set of goals and objectives 
adopted by the department 


s 




w 


2) Knowledge and use of emerging curriculum trends (validity 
issues) 


s 


W 




3) Curriculum that addresses the full range of student effectiveness 
issues, both current and future 


sw 


W 




4) Objectives which set the framework for operation of the system 
and its sense of priorities 


s 


W 




5) Demonstration that the system is contextually responsive to 
national, state, and other expectations as evidenced in local 
initiatives 


s 


W 




6) Major programmatic initiatives designed to be cohesive 


s 


W 




7) Provision of explicit direction for the department head and 
professional staff 


sw 






8) Evidence of comprehensive, detailed, short and long-range 
curriculum management planning 


s 


W 




9) Mechanisms that exist for systemic curricular change 


s 


W 





ERiC 
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Standard 3 

Connectivity and Equity: The department has documentation 
explaining how its programs have been developed, implemented, and 
conducted Common indicators are: 


Adequate 


Partially 

Adequate 


Inadequate 


1 ) Documents/sources that reveal internal connections at different 
levels in the system 
S: Present in catalog 
W: N/A 


S 






2) Predictable consistency through a coherent rationale for content 
delineation within the curriculum 
S: Try to emphasize the ADDIE model throughout curriculum 


S 


W 




3) Equity of curriculum/course access and opportunity 


sw 






4) Allocation of resource flow to areas of greatest need 


sw 






5) A curriculum that is clearly explained to members of the 

teaching staff, department head, and other supervisory personnel 


sw 






6) Specific professional development programs to enhance 
curricular design and delivery 
S: This comes through the college. 


s 




w 


7) A curriculum that is monitored by department personnel 


sw 






8) Teacher and administrator responsiveness to department 
policies, currently and over time 


sw 







Standard 4 

Feedback: The department looks for results from department 
designed or adopted assessments to adjust, improve, or terminate 
ineffective practices. Common indicators are: 


Adequate 


Partially 

Adequate 


Inadequate 


1) A formative and summative assessment system linked to a clear 
rationale in department policy 

S: Completed through individual course evaluations, 5-7 year 
assessment, and exit evaluations. 

W: This is accomplished through the NCATE evaluations. 


s 


W 




2) Knowledge and use of emerging curriculum and program 
assessment trends 

S: Things will need to be fixed to accommodate the change to 
semesters. 


s 


SW 




3) Use of a student and program assessment plan which provides 
for diverse assessment strategies for multi-purpose at all levels— 
college/department/classroom. 


w 


S 




4) A timely and relevant base upon which to analyze important 
trends in the instructional program 


s 


W 




5) A way to provide feedback to the teaching and administrative 
staff* regarding how classroom instruction may be evaluated and 
subsequently improved 


s 




w 


6) A vehicle to examine how well specific programs are actually 
producing desired learner outcomes 


s 




w 


7) A data base to compare the strengths and weaknesses of various 
programs and program alternatives, as well as to engage in 
equity analysis 




S 


w 
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8) A data base to modify or terminate ineffective educational 
programs 

S: Course offerings are driven by enrollment. If students don't like 
the course, they walk. There is some concern about duplication of 
material/content The department relies on students for feedback. 
The system in place is to complain to the department head, who will 
take appropriate actioa 


s 




w 


9) A method/means to relate to a programmed budget and enable 
the system to engage in cost-benefit analysis 


s 




w 


10) Organizational data gathered and used to continually improve 
system functions 


s 


w 





Standard 5 

Productivity: The department has been able to improve productivity. 
Common indicators are: 


Adequate 


Partially 

Adequate 


Inadequate 


1 ) Planned and actual congruence among curricular objectives, 
results, and financial costs 


S 


W 




2) Support systems that function in systemic ways 


S 




w? 


3) College and department climate conducive to continual 
improvement 


sw 






4) Specific means that have been selected or modified and 
implemented to attain better results in the department over a 
specified time period 


s 


W 




5) A planned series of interventions that have raised student 

performance levels over time and maintained those levels within 
the same costs parameters as in the past 
S: He believes that the department feels that the students are getting 
better. This is due in part to the monitoring of admissions. 


s 




w 


6) A financial network that is able to track costs to results, provide 
sufficient fiduciary control, and is used as a viable data base in 
making policy and operational decisions. 

S: There are limits to this tracking. Decisions are made mostly to 
accommodate the most pressing needs. "The place is band-aided 
together" because that "is the name of the game." 


s 




w 


7) Facilities that are well-kept, sufficient, safe, orderly, that comply 
with all requirements, and facilitate delivery of the instructional 
program. 


sw 







General comments about the curriculum: 

Dr. Smellie: 

The department has good facilities. This is due in part to his involvement in the design of the 
building and in his efforts to increase the square footage assigned to the department 

Do we have holes in the new curriculum? We haven't had time to test it yet. 

Do the masters students have enough development experience? The department may be shifting 
too much towards theory for the masters program. 

It is important to make sure we teach applied elements in the masters program, since graduates 
could be the only person hired and need to be able to run a shop for a small company. This approach has 
built the masters program into what it is today. The masters program focuses on development more than the 
Ph.D. orEd.S. programs. 



Q 
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The department does have a strategic plan for improvement. This plan outlines some of the core 
beliefs, philosophies, and values in the department. It also includes a mission and vision statement, along 
with strategies for achieving these statements. (Parts of the 1995 plan are included in Appendix C.) 

Dr. Walcott: 

There could be better articulation of the curriculum. Currently there is not scope and sequence. 
Future efforts will be made to collect a current list of syllabi and goals for all courses and communicate this 
information to everyone in the department. 



Appendix B: Curriculum Matrix 
Curriculum Audit 



Theory Practice 

M D M D 



Design 


6080 


EdS/PhD 


6510 






6250 










6260 


7140 








6500 


7150 








6460 


7160 








6480 


7170 








6490 


7180 








6810 


7190 








6870 


7200 






Development 


6460 




5230 


7840 




6480 




5240 






6490 




6210 










6220 










6450 










5400 










6800 




Management 






6470 

BISE 6350 


7820 


Utilization 








7830 


Evaluation 






6510 


(EdS) 
PSY 6010 
ED 6570 
7810 
7960 










(PhD) 
ED 7310 
7850 
7860 
ED 6600 
ED 6610 
ED 6770 
7810 
ED 6010 
ED 7700 
ED 7780 


Research 




7200 


6510 


7810 
ED 6570 
ED 6600 
ED 6610 
ED 6770 










7810 




* 






ED 6010 
ED 7700 
ED 7780 


Ethics 








6080 


Other (Thesis, 






6800 


7970 


Dissertation, 






6940 


7960 


Etc.) 






6950 










6960 










6970 







Curriculum Audit Class List 
M.S. 



Theory 

Design : 

6080-Instructional Technology Core 
6250-Instructional Design Theory 

6260-Learning and Communication Theories in Instructional Technology 

6500-Instructional Development Tools 

6460-Distance Education 

6480-Instructional Simulations 

6490-Instructional Technology in Adult Education 

6810-Independent Research 

6870-Current Issues 

Development : 

6460-Distance Education 
6480-Instructional Simulations 
6490-Instructional Technology in Adult Education 

Practice 

Design : 

65 10-Research and Evaluation in Instructional Technology 
Development : 

5230-Instructional Graphic Production 
5240 -Producing Distance Education Resources 
6210-Digital Audio- Video Production 
6220-CBI Authoring Tools 
64 50 -Instructional Product Development 
5400-Computer Applications 
6800-Project in Instructional Technology 

Evaluation : 

65 10-Research and Evaluation in Instructional Technology 
Research : 

65 10-Research and Evaluation in Instructional Technology 



Other : 

6800-Project in Instructional Technology 
6940-Plan C Internship 
6950-Plan C Externship 
6960-Plan C Creative Project 
6970-Plan A Thesis or Plan B Paper 




EdS/PhP 



Theory 

Design : 

7140-Designing Instructional Systems 
7 150- Advanced Instructional Design Theory 
7 160- Advanced Learning Theory for Instructional Technology 
7170-Designing Instructional Technology Tools 
7 180- Advanced Techniques for Instructional Technology Production 
7200-Advanced Research in Instructional Technology 

Research : 

7200-Advanced Research in Instructional Technology 

Practice 

Development : 

7840-Instructional Product Development Practicum 
Management : 

7820-Funding proposal Practicum 
Utilization : 

7830-Instructional Product/Research Review Practicum 

Evaluation : 

EdS : 

PSY-60 10-Program Evaluation in Psychology & Education 
ED-6570-Introduction to Psychology and Educational Research 
7810-Research Seminar 
7960-Practicum 

PhD : 

ED-73 10-Teaching and Learning Foundations 
7850-Instructional Evaluation Practicum 
7860-Instructional Empirical Investigation Practicum 
ED-6600-Measurement, Design & Analysis I 
ED-66 10-Measurement, Design & Analysis II 
ED-6770-Qualitative Methods 
7810-Research Seminar 

ED-60 10-Program Evaluation in Psychology & Education 
ED-7700-Single Subject Methods & Design 
ED-7780-Qualitative Methods II 

Ethics : 

6080-Instructional Technology Core (needed if holds degree in a field other than IT) 
Other : 

7970-Doctoral Dissertation 
7960-Practicum 
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. Appendix C: Instructional Technology Department Strategic Plan (October 20, 1995) 



Introduction 

The Department of Instructional Technology intends to be continually involved in goal definition, 
revision and assessment. This should be an on-going process based upon state and national needs and 
trends. Two important assumptions should apply to this strategic plan. 

1) The Instructional Technology Department views this plan as on-going in nature, but still providing 
direction for the department. 

2) Something ought to happen as a result of the strategic plaa 

Values and Core Beliefs 

The faculty has reviewed the values and core beliefs from the university strategic plan and believe 
they represent the values and beliefs of the department faculty with some minor modifications. These 
values and beliefs are as follows: 

Learning 

• Development: We foster the opportunity for intellectual, physical, social, moral, and cultural 
development of the whole person. 

• Discovery: We encourage research activities that bring recognition to the College and University and 
contribute to the body of knowledge in the field of Instructional Technology. 

• Creativity: We seek creative solutions to problems in the learning process. 

• Debate: We are open to challenge and debate in our learning environments. 

Openness 

• Access: We facilitate access to academically qualified graduate students with an appropriate 
student/faculty ratio. 

• Friendliness: We display a friendly, courteous, and helpful attitude. 

• Tolerance: We tolerate all people regardless of background or race. 

• Graciousness: We give students, faculty, and staff our individual attention. 

• Diversity: We seek applicants for graduate study from groups who are under-represented or who have 
suffered discrimination. 

• Humor: We smile, laugh, and enjoy. 

Citizenship 

• Democracy: We share in the governance of the College and University, both in determining our goals 
and in shouldering our responsibilities. 

• Responsibility: We account for ourselves and our stewardship of public trust. 

• Partnership: We believe that all employees and students in the program play an important part in 
fulfilling the Department, College, and University's mission and work. 

• Ethics: We strive for the highest feasible standards in all our endeavors. 

• Respect: We offer civility in dealing with one another and reject all forms of rudeness. 

• Equity: We strive for equal treatment of all members of our diverse department and of students and 
scholars. 

Service 

• Relevance: We offer teaching, research, and service that are well organized, informed, and relevant to 
the profession. 

• Communication: We strive for open and frequent communication among the students and faculty. 
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• Improvement: We never cease in our desire for high standards and a "cutting edge program." 

• Flexibility: We strive for change and constant improvement that enables a prompt and reasonable 
response to needs in the field of Instructional Technology. 

• Practicality: We offer programs that are well-conceived coherent, up-to-date, and centered on the 
needs of the profession. 

• Credibility: We respond to the needs of our professional community while striving for the highest 
feasible standards in each of our endeavors. 

Productivity 

• Entrepreneurship: We strive for a department environment that nurtures disciplined creativity, 
innovation, organization, and productivity. 

• Efficiency: We continually encourage better ways to achieve our goals. 

• Teamwork: We solve complex problems by working as a team contributing our expertise and 
knowledge to create solutions to important problems. 

• Cooperation: We enhance the department through team efforts. 

• Quality: We seek quality over quantity. 

• Recognition: We celebrate the achievements of students, faculty, and staff. 

• Commitment: We are committed to the land grant mission of Utah State University to foster 
intellectual development and to meet instructional technology degree needs of the state. 

Vision Statement 

To be an exemplary leader in scholarship, research, invention, development and practice in the field of 

instructional technology. 



Mission Statement 

The mission of the Department of Instructional Technology is: 

• To understand through scholarship, research, invention, development and practice how to select and 
organize information and instructional materials to enable learners to acquire knowledge and skill in 
the most effective, efficient, and appealing manner. 

• To interact in a collegial environment with faculty, students and professionals in scholarship, research, 
invention, development, and practice. 

• To disseminate theory, research, products, procedures and practices to students, the profession, 
education, business, industry, and government 

• To prepare students. 

As a consequence we hope to be recognized for exemplary leadership in research, design, development and 

academic programs in instructional technology. 

Strategies to accomplish the Vision and Mission 

• We will develop an undergraduate multimedia development minor. 

• We will refine the undergraduate library media teaching minor. 

• We will refine the Masters Degree program track in Educational Technology. 

• We will develop an Instructional Technology Ph.D. program if the core for the existing IDP does not 
meet the needs of Instructional Technology Ph.D. students. 

• We will develop programs for delivery over electronic delivery systems (e.g. ComNet, EdNet, 

InterNet, On-site, Campus). 

• We will develop new admission standards and recruit better, not more students. 



• We will increase the number of jointly published articles. 

• We will attend and make joint presentations (with students) at multiple professional conferences and 
report back to the faculty things learned. 

• We will make better use of graduate students to teach basic courses. 

• We will make better and smarter use of our existing space and find space for Ph.D. students to be 
housed in the department. 

• We will increase grantpersonship. 

• We will improve the library collection related to our field. 
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Mail and Email Survey 



Prepared by: 
Richard Harmon 
Jacques du Plessis 



I. Sampling Methods 

The purpose of this needs assessment survey was to contact recent master's graduates and 
measure their perception of the relevance of a number of ISD related skills and topics 
with regards to their current job and USU's corresponding success in preparing them in 
these specific skills and topics. 

A contact list was generated from available documents comprised of IT department 
master's graduates since 1994. Approximately 1 10 letters and surveys where mailed out 
giving respondents an opportunity to reply via mail or through an on-line survey. An e- 
mail letter was send to an additional 20 respondents requesting their response through the 
same on-line survey. This created total potential sample size of 130. 

II. Limitations 

42 respondents replied before the deadline to our survey request giving us a 32% return 
rate on the graduates surveyed. While this may limit our ability to generalize our data to 
the whole population of graduates during the years polled it does give us enough data to 
run meaningful descriptive statistics revealing possible strengths or weaknesses in the 
department for the years surveyed. 

III. Results 

The following table illustrates and summarizes how the survey instrument measured 
training gaps or excesses in specific areas related to the entire ISD process. Each question 
was rated on a 1 to 5 point Likert scale. Mean scores from each "current importance" 
question where compared to mean scores of each corresponding USU "preparation" 
question. Variances are shown to document the variability in response for each question. 

The following sample question illustrates how each topic was rated to measure how 
relevant a given topic/skill is to their current job and how well USU prepared them in that 
given topic/skill. 



Technical writing/proposal writing 



Analysis and Design Importance to current job. Mean scores How well 
USU prepared you. Mean scores Variability of answers (Variance) 

Needs assessment and task analysis 4.19 3.90 ,83/.96 

Content analysis and audience analysis 4.38 3.91 .737.94 
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Objectives writing, script writing, instructional strategies, learning theory 4.38 3.88 

,82/.97 

Content sequencing, storyboarding, resource specification, message design 3.88 

3.69 1.02/1.00 

Development, Implementation and Evaluation 

CBT authoring (Toolbook, Authorware, Director, etc.) 2.95 3.29 1.38/1.09 

Web authoring (HTML, Dreamweaver, Front Page, Homepage, Javascript, etc.) 3.07 
2.57 1.33/1.09 

Digital and print-based resource creation (graphics, audio, video, animation, etc.) 3.29 
3.36 1.29/1.16 

Instructional product or program implementation 3.64 3.33 1.16/1.00 

Product and program evaluation - formative/summative (design, analysis, reporting) 

3.69 3.40 1.09/. 89 

Other Categories 

Project management and involvement in the entire product development life cycle 3.90 
3.14 1.12/1.12 

Technical writing/proposal writing 3.90 2.86 1.05/1.12 

Dealing with colleagues and learners from other cultures and backgrounds 3.90 3.33 

1.05/1.16 

Presentation skills (planning, speaking, media preparation) 4.14 4.33 1.05/.75 

Instructional management systems 3.57 2.81 .96/1.13 

Automation of instructional design and develoment 2.98 2.81 1.22/1.37 

Data in bold are highest of compared measures 

Four open-ended questions where also asked seeking input on what they felt were USU 
program strengths, weaknesses and areas requiring future student preparation. To gauge 
the contextual setting of the respondent's answers we asked them to give a brief current 
job description. We also asked the respondents to identify three of their primary sources 
for staying current in the field. 

The first question asked what they perceived to be the strengths of the program. 
Presentation skills were mentioned most often as the most valuable skill, followed by the 
instructional design component. Some othre issues mentioned infrequently were 
teamwork, computer skills, and project management. 

The second question asked what they perceived to be lacking in the program. 

It was not as easy to group and interpret these results since that might slant the 
perception. Following are some key words, phrases or sentences from this section. 
Hopefully this will more accurately reflect the views of the respondents. 

17. 1 can't think of an area that was lacking. 

17.1 would have liked more application of the different learning theories. 

17.1 don't feel there was anything lacking. 

17.1 really don't feel to comfortable in web development, but that wasn't included as a 
part of my program. 
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17.1 think there needs to be more learning theory. But not just the theory. It needs to be 
applied to real life situations. With out the appropriate application is very difficult to 
know what situations require which theories. 

1 7.1 feel the department or faculty could have a greater level of contact with industry to 
get a feel for what they are expecting from graduates in the private sector and prepare 
accordingly. 

17.Project management. 

1 7.1 needed more technical writing experience. 

17.There was little available training in how to do development and very limited training 
in applying learning and instructional theories to real design and development tasks. 
Additionally, there was very little opportunity to study instructional theories other than 
Merrill's CDT and transaction theory. 

17.Not enough emphasis on multimedia skills... strong computer skills and competencies 
are still greatly needed in this field. Portfolio creating is essential in landing a good job. 

I would suggest a summer class at the end of the program that is a career prep and trend 
class that builds portfolios. Also, students should learn how to take VERY complex 
information and translate it into plain English in a Web-based learning or CBT product. 
17.Technical skills 
17. Writing Skills, Business Etiquette 

17. Project management classes were very weak. Most of the CBT development skills I 
took out of the program came from my own independent study courses. The department 
could do a better job of teaching how to develop for different delivery modes (ILT, CBT, 
paper based, etc...) 

17.Development skills must not be phased out. The department is becoming a face for 
educational psychology. I would not be as prepared for my job if I started the program 
now. Production/multi-media skills must be the TOP priority for new professor 
candidates. When Dr. Soulier retires, we will lose a core strength of the reputation of 
"hands-on" that USU is known for. 

17. Project management is the biggest area that sticks out. I got nothing out of the course 
I took in project management. Technical writing is another area. I think all students 
should be required to take a course in technical writing, it is a critical skill in most 
instructional design jobs that I have seen. The front end piece is another area I found to 
be weak. An entire course on front end analysis is needed. The other area that is weak in 
the program is in the area of evaluation. I didn't even learn about Kirkpatrick’s four 
levels of evaluation in the program. 

17.1 spent most of my last year of the program trying to see a way that I was going to use 
the information I was learning in a job I felt I would enjoy. I also felt that program lacked 
an emphasis in implementation. The theories we were being taught were not even being 
utilized on us. But that aside, I still feel there could have been more project based classes 
that taught us how to assemble to right team to create real world instructional products. 
Most of the knowledge base I use today aside from theory and analysis, I learned from 
classes that I performed as independent study. I learned Photoshop, Director, HTML 
editors, Database, Programming languages, tracking, assessment, multimedia 
development, animation and script writing software on my own. Dr. Soulier attempted to 
give an overview of some of these tools, but if I were to try and get a job from skills 
learned in that first year class in the high paced product development jobs I have worked 
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in, I would have starved to death. 

Consulting: Attract more recruiters from other top consulting firms who are hiring 
Instructional Designers, trainers, content developers, etc. Help the students prepare for 
these interviews and let them know of possible career opportunities they will encounter in 
the present industry. I know this has improved since I left the department, but I don't 
remember a single visit from Anderson Consulting, Ernst and Young, Bain, Boston 
Consulting Group, McKenzie and Co., Worthlin Consulting etc. 

Stand Up training: Offer more than one class that teaches speaking and presentation 
preparation skills. Try and create opportunities or provide the resources to learn the 
technologies that facilitate group instruction, tracking and management. Some of the 
most exciting and challenging jobs in the country right now are at training centers that 
use very sophisticated and expensive presentation rooms. Our graduates don't have a 
prayer being hired at those location due to their lack of preparation on the requisite 
software or technology. 

Theorists and Research: I think this is the strength of the program, but creating a program 
that guarantees employment in research projects upon acceptance is an obvious 
shortcoming of the department. I was employed for most of the two years that I was 
there, but it was due to my own efforts in almost every case. 

Content Development: This is an area the department has spent considerable money and 
resource on, and I know has tried to provide the students with the machines and software 
they need to work on projects and class work. Don't let the technology fade into 
obsolescence. The demands on IT employees to know how to use the machines and 
software to develop their own products increases each day. They at least need to have a 
good idea of what is involved in creating an animation, authoring component or web- 
delivered course. We have entered into an era of client/servers, Internet based/delivered 
education, and complex learning systems. Unless the IT graduate wants to write manuals 
the rest of their life, they will need to up to speed on close to 20 different computer 
programs, and at least two to three different authoring systems. 

Technical writers: For those that are going to focus on print-based assessment, job-aids 
and instructional tools, there was nothing in the program that taught them how to use 
Quark, PageMaker or more importantly FrameMaker. Also nothing taught or addresses 
the theories behind on-line tutorials or help systems, the softwares that are used to create 
them. There was no examples or non-examples of good instruction or help. Tool 
Builders: Many of the people that I still have contact with such as Thor Anderson, 
presently with Oracle, are interested in tool development. They are not interested in re- 
creating ToolBook or Authorware or Quest. They want to create the holy grail of 
instructional tools - a tool that will allow content to be separate from the tool. These tools 
are actually a combination of components or building blocks that can be interchangeable 
based on the type of learning that is done. How wonderful would it be to attract forward 
thinking tool-builders to the program to increase research and development of tools that 
reduce the development cycle and finally deliver on the promise that authoring tools have 
fallen short in - rapid product development. 

Instructors: This may have also changed, but there is very little opportunity for students 
to teach what they know, so that they are able to develop the talent and germinate the 
seeds of professorship. Provide opportunities to encourage research, academia, and 
knowledge sharing. This was done to some extent with symposiums, brown-bags, and 
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workshops, but i think more can be done to involve more of the students. Students who 
are forced to work at unrelated jobs to stay in school or feed their families. 

17. At the time there was very little equipment and software to develop with. This has 
since changed but should be kept up to date. The instruction related to development was 
ok but could have been much better. I didn't care for two of the main instructors being 
inaccessible and gone a lot. I felt like the program was really well known at the time I 
came into it but that new research and cutting edge ideas were lacking soon after. Kind of 
resting on their laurels. The drive that had built up the department had disappeared. 

17. Lack of effective teaching practices by certain professors: course design did not 
reflect practices and theory of IT that they were teaching Bias towards "hardware" 
aspects versus theory: e.g., learning theory (common comments by students related to less 
need for theory and more need for courses relating to computer-based tools). More focus 
and depth in learning theories (including practical applications), effectiveness of teaching 
tools and methods, etc. is needed Acceptance of more graduate students than could be 
reasonably managed by professors (class size, number of advisers/professor) Enhanced 
discussion of how IT differs from many similar disciplines and degrees, including 
education. 

17. Programming skills 

17.Technical writing and proposal writing. Web authoring. 

17. The focus on using computers for instruction was also a liability. The scope of my 
education needed to be broader. I've since learned that many people don't use only 
computers to create study aids, etc. Also, evaluation plays a much bigger role in 
instructional design than was implied during coursework. In order to "cover" oneself, if 
not for the quality of the product, evaluation must be emphasized. We profess to design 
effective instruction, but how often are we perhaps running behind schedule and cut truly 
well-planned and documented evaluation. As designers, and especially as educators (isn't 
that what we are before anything else? Our products are designed to educate), we need to 
emphasize evaluation. 

17.1 noticed that there was not a lot of direction provided for the student. You have your 
1-2 year agenda as far as the classes needed for graduation, but I heard several complaints 
about no direction. I feel a bit different about the matter. I feel that the student needs to 
take the bull by the horns and make it happen. I learned that questions and interviewing 
never cost me anything. Students need to be proactive in seeking out education. 
Education is not a one way road. Your multimedia IT area is quite week. Especially 
when most all students wanted exposure to multimedia. I found in within the ART Dept 
at USU, and fulfilled my needs there. Once again... taking the bull by the horns. CBT 
and digital production are very important in the field and have proven to be vital skills for 
me to have. Not all jobs within the field have need for hard-core production skills, but 
knowledge in the area is a MUST! ! ! ! I have a great job due to my digital production 
skills among other things. One thing that might help is to create some canned proposed 
programs, classes to take etc., for those students that don't have a clue what they can do 
with the knowledge that the program can give them. Maybe this will give students more 
of an idea what Instructional Technology is and what you can do with it etc. I know that 
this was tried with IT common foundation, but still some work could be done there. Just 
some suggestions 

17. For me, the program was exactly what I needed. It emphasized the current and future 
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trends in training and let me sample a bit of each area. It gave me the freedom, though to 
design my education in such a way that I could pursue the career I wanted in IT. 

17. Analysis (needs, task, content sequencing, objectives, storyboarding, message design), 
Assessment (strategies, tools, correlations with objectives, ROI...) 

17. A look at other instructional paradigms, using other innovative ways to design 
instruction instead of the traditional Analysis, Design, Development, Implement, 
Evaluate. 

17.1. Tools: I regret not taking more tool classes (e.g., Director, AuthorWare). 2. 1 would 
have liked more learning theory (but I obtained that through the doctoral program). Quite 
honestly, the master's program prepared me more for teaching than did the doctoral 
program. I wish the current doctoral program had been in place when I completed my 
doctoral coursework. 

17. More information on emerging technologies for instruction. (IE: internet or whatever 
the medium-du-jour happens to be that industry is embracing at the time) More classes 
involving the design and development of interactive courseware would be very helpful. 
(Internet /ICW/CBT, etc.). 

17.More classroom curriculum development (non computer-based). 

The third question asked them what future IT trends do the perceive that we need to 
prepare students for. 

The response to this question was fairly uniform. The Internet! Web-based training, 
distance education, and in the words of one of the respondents: "CBT is going to the web 
faster than anyone can keep up with it." Other skills mentioned related to this 
environment, including skills with specific software programs like Authorware, 
Dreamweaver, 3D Studio Max, Premiere, Illustrator, Java, HTML, VRML, and 
Photoshop. 

The fourth question asked them to give a description of their current job. 

Most respondents were employed as trainers, teachers, consultants, and instructional 
designers. 



IV. Conclusions 

The data reveals one obvious strength and two obvious weaknesses in the program. 
Respondents rated presentation skills at above average importance (4.14) and they rated 
USU's preparation in this skill (4.33) equal to or greater than their perceived need. 
Technical writing skills were rated at above average importance (3.90) but USU's 
preparation in this skill was far less (2.86) than their perceived need. Web authoring skills 
were also rated at above average importance (3.07) but USU's preparation in this skill 
was significantly less (2.57) than their perceived need. 

From the data it is apparent that front-end analysis and design is considered of high 
importance to recent master's graduates with USU preparation in this are nearly equal to 
the perceived need but falling slightly short in each category. The smaller variance of the 
responses related to front-end analysis gives weight to the consistency of the perceptions 
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amongst respondents. 

All remaining skills and topics except two showed the respondent's perception of 
importance to their current job exceeding their perception of the adequacy of their 
training in that skill at USU. Only in CBT authoring and digital resource creation did 
their perception of USU preparation exceed their perception of its importance in their 
current job. 

The skill rated as the least important in their current job was CBT authoring. However 
this same rating had the highest variance demonstrating the broad range in responses 
from those surveyed. The skills rated the most important where front-end analysis and 
presentation skills. 

V. Future Needs 

Several specific items should be addressed in future needs assessments. First, Phd and 
EdS. graduates should be polled as well because their skill sets are usually different than 
masters students. Also, questions regarding what skills or experience graduates feel 
helped them get a job would be of value to future graduates. 

IV. Raw data packet. 

(see the attached file "rawdata.txt") 
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Instructional Technology 7010 

David DeBry, Tom Nickel 
April 24, 1999 

Telephone Survey of Instructional Technology Departments 



I. Sampling Methods 

This survey was conducted in order to provide an external frame of reference for the 
departmental Needs Assessment. Findings and determinations, which come from the inside, 
have to be considered as the priorities. Input from other IT departments allows us compare and 
consider our own conclusions within a broader context, and to notice and consider any major 
divergences from important trends or patterns. 

As a checkpoint and a source of ideas, there was no need to select our subjects at random and 
ask easily quantifiable questions. In fact, we did just the opposite. We purposely chose most of 
the larger and well-known Instructional Programs in the country, with a few of the smaller 
programs added to the mix. In almost every case, we were able to speak with the Department 
Chair. 

The interviews themselves were carried out by two interviewers, using the same set of 20 open- 
ended questions (see appendix). The initial question in a given area could lead to further 
probing questions. The overall tone was conversational, and the average interview length was 
one-half hour. Some portions of the questionnaire were more fruitful than others. The 
completed interviews themselves, eight of them, are included as appendices. 

II. Limitations 

In designing and carrying out this study, we saw no limitations other than the fact that we 
interviewed only eight institutions. In doing this, we may have missed some good, solid ideas 
from other programs which could have been implemented into our program, with great benefit. 

III. Results 

The remarks below summarize the insights gained in the most substantive areas of the survey as 
a whole. 



1) The ISP Process 
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Despite changing times, the ISD process is alive and well. Other programs are adding new 
perspectives to the process in later, upper level courses, but it forms the heart of almost 
every program. One respondent stated that their graduates claim a systematic ISD process 
is the most important thing they took from school to their job. Concentration on theory, 
process and practice was also stressed by a number of programs. 

However, after the core courses, students are exposed to a variety of different planning 
perspectives which are significantly different than the ADDIE model. One program was 
most strongly influenced by Human Performance Technology, another by a more problem- 
based approach. Still another was developing a new emphasis on teacher education, 
which brought in very different sorts of planning models. 

2) Tool Classes 

Most respondents claimed that this is a particularly difficult area to gain agreement on as 
far as its role in the program. But interestingly, almost everyone interviewed was in 
agreement - tool courses should not be taught in isolation from instructional design. Most 
departments incorporated projects into class work which required the use of tools. There 
was also widespread agreement as to the importance of graduates developing skills in some 
of the tools used in the field, including authoring tools, programming languages, and web 
software. Some of the programs were putting a strong emphasis on tools for producing web- 
based courseware, and all programs were teaching web development in one way or 
another. 

3) Organization Development/Change 

Very few programs are able to do as much in this area as they would like to; some site it as 
their biggest weakness. Most had classes dealing with project management, but beyond 
that not much is taught. It was the opinion of one department that a project is handed off 
to someone else for implementation so they don’t teach that. Some of the larger programs 
have been able to develop courses of their own, where they focus on responding to system 
change and diffusion of innovations. Organizational change theories are part of the 
Comprehensive Exams for one program. 

4) The Net 

Most, but not all, of the programs in the survey are beginning to emphasize the Web, 
teaching Web-based tools and offering courses in instructional design for the Web. Most 
programs also use the Web to supplement classroom activities in some courses. 



Strangely enough, although some of the universities in which these IT departments are 
located are producing large amounts of Web courseware, the IT people themselves are not 
involved in the university-wide activity much at all. Florida State and BYU were the only 
programs where they were involved in developing on-line couresware. One of the 
respondents offered the opinion, "... the Web has caused a deterioration of the design 
process. It's replicating all the worst points of face-to-face instruction." The image these 
interviews produced for us as IT departments very slowly integrating the Web and producing 
material for online delivery, while someone else is translating enormous amounts of textual 
material to html pages inside courseware containers such as Web CT. 



With a few exceptions, IT doctoral dissertations are evolving rapidly toward an equal 
mixture of quantitative and qualitative research paradigms, sometimes within the same 
dissertation. There was only one program in which qualitative research was frowned on -- 
but then there was also one program in which qualitative research now clearly 
predominates. Masters students are not always required research classes, but doctoral 
students are. 



Respondents reported that Masters students graduate almost without exception into 
corporate positions or work as independent developers, although some graduates go back 
into public education. Even doctoral graduates were split 50/50 between business and 
academia in most cases. Many departments recommend the academia track for doctoral 
students, but don't require it. 

One program integrated the many alumni working nearby into a course in "Issues in 
Training." It is an extremely useful and popular class, with the real-world job experiences of 
recent graduates as the framework. 



5) Research 



6) Alumni 



7) Strengths 



Diversity of opinion and perspectives 
Serving needs of graduates 
Teacher education 
Good facilities 
Small classes 



5 

2 

1 

1 

1 
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Instructional Television component 



1 



8) Weaknesses 



Lack of organizational theory 
Not enough theory 
Weak on production 
In transition 

Organization of Ph.D. program, relating 



1 



1 



1 



1 



theory to research to practice 
Financial support of students 
Need more advanced courses / seminars 



2 



2 



2 



IV. Conclusions: 

These conclusions are based on findings from other institutions and may already be in place at 
Utah State University. 

1) Integrate tool courses and instructional design courses, over a multi-course sequence if 
necessary. Continue to require tool courses, but only teach them within an instructional 
design framework. 

2) Define appropriate Organizational Change or Management Theory courses for Instructional 
Technology students and build them into the required curriculum. Work with other 
departments if the department itself is unable to offer them without support. The 
overwhelming majority of IT students will be working in a corporate environment. A 
complete view of IT includes the context in which it is practiced. 

3) Determine processes and procedures through which an instructional design perspective can 
be brought to bear on courseware development for the Web on a university-wide basis. 

4) Find ways to involve recent graduates working nearby in certain courses. Let them teach or 
co-teach a course, under faculty supervision, that is designed to work with other elements 
referred to in (2), above. 

5) Secure more funds for graduate students. 

6) Create some sort of partnerships with departments around campus so that class projects can 
be done for these departments. 
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V. Future Needs 

In administering these interviews, most of our questions were quite broad and there was not 
enough time to elaborate on some important points. Our suggestion would be to look at the 
programs interviewed, take one or two good and get specifics on how to implement these 
ideas. 

For example, Arizona State University has recent graduates teaching a course. We would 
recommend that we further pursue how it is that this is being done, which is something we 
couldn’t do within a half-hour interview. 
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USU IT Department Needs Analysis: 

Focus Groups/ Follow-up Questions 

Prepared by: 
Jim Rogers 
Joel Duffin 



I. Sampling 

The goal of conducting focus groups was to gather information on what companies who hire 
Instructional Designers look for in the people they hire. For this purpose we: 

SLC Focus Group 

1 . Identified companies in the Ogden - SLC - Utah Valley areas who hire instructional 
designers by speaking with Professors, Alumni Lists, and talking with the SLC ISPI 
president and other employers. 

2. Set a time and a place for a meeting. 

3. Contacted people within the identified companies who do the hiring of instruction designers. 
We invited these people to come participate in the focus group. We told them that we would 
provide dinner and take about one hour and half. In addition we sent participants a list of the 
framing questions of the focus group. 

4. Followed up. 

We initially contacted 20 companies and ended up getting representatives to participate from the 
following 6 companies: TenFold Corporation, Novell, American Stores, Allen Communications 
Inc., Utah Transit Authority, CES Department of the Church of Jesus Christ of Latter Day Saints. 

Phone Call Follow-up 

The purpose of the phone call follow up was to validate the information gathered during the 
focus group. We targeted similar populations. In doing so, we: 

1 . Identified a list of possible companies and contacts by speaking with Department Chair and 
Staff Assistant. 

2. Researched phone numbers and e-mail addresses for contacts. 

3. Attempted to contact the individuals through voice and e-mail. 

(It should be noted here that finding this information and attempting to make the contacts was 
very time consuming without many results. Many messages were left unanswered). 

4. Out of our list, three contacts were established with the following companies: IEC, 
California; Arthur Andersen, Chicago; and Ryder Trucks, Colorado. 



II. 



Instrumentation 



SLC Focus Group 

We began the focus group by eating dinner and informal discussion. After four participants 
arrived we explained that we would be tape recording the meeting and video taping portions of 
it. After turning on the tape recorder we asked participants to introduce themselves to the group. 
After introductions one facilitator wrote the framing questions on a white board and invited the 
group to respond to the first question. At the outset of the discussion the facilitators intervened 
with follow up questions. After a while, participants began to key off of and regulated the 
discussion themselves. In addition to tape recording and video taping the discussion, facilitators 
took notes. 

Phone Follow-up 

Once contact was made by one of the researchers, the general guidelines for the focus group 
were followed. In general, the participants were given the open ended questions and allowed to 
respond. In addition to the framing questions, specific probing questions were also posed so as 
to validate issues that were raised during the focus group discussions (i.e. ‘Do you feel 
teamwork is an important skill?’). During the interview, the researcher took notes. 

Framing Questions 

We used the following questions to guide the discussion: 

1 . What specific skills and experience do instructional designers need to succeed? 

2. What is the difference between and expert and novice instructional designers? 

3. What do instructional designers do at your company? 

4. What are the biggest weaknesses you see in recent graduates who you hire to instructional 
designer positions? 

5. What skills will instructional designers need to succeed in the future? 

III. Limitations 

The major limitation that applies to the data reported is that the SLC focus group brought in 
companies mostly in the SLC area. They may not be representative of all companies (nation and 
worldwide) that hire Masters Degreed Instructional Designers. Although the phone follow-up 
interviews were designed to address this limitation, the fact that only three contacts were did 
little to lessen this threat. The fact that the telephone responses were highly similar to those of 
the focus group provides us more assurance that these results are valid. 

IV. Results 

Results are reported according to the questions that they respond to: 

What specific skills and experiences are necessary for instructional designers? 

1. ISD Process 
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It was clear from the focus group and follow-up phone calls that an important part of an 
instructional designer’s knowledge should include the ISD process. One participant said “If ask 
asked a perspective employee to do any part of analysis and design they should be able to do it 
without looking at the book”. In addition, it was also important that instructional designers are 
effective functioning within the model. Many of the focus group participants echoed similar 
sentiments, discussing the difference between knowledge of theory and process. Many thought 
that recent graduates should be as well versed in process as in theory. They explained that the 
theory is often in place in many companies. 

Although the telephone participants included knowledge of ISD as an important skill, two of the 
three indicated that a practical knowledge of the process was more important than a mechanical 
understanding. One participant said instructional designers “must be versatile enough to follow 
it (ISD process) rote or be able to deviate”. Thus it is clear that knowledge of the ISD process 
be situated in authentic practice. 

2. Teamwork 

There was general consensus amongst many of the participants that teamwork and more 
specifically, being able to work as part of a team, is an extremely important skill. This skill 
could be defined as narrowly as working with one person, usually the subject matter expert. “We 
cannot expect someone to be a subject matter expert because every environment is different” 
was a statement that indicated the importance of being able to work cooperatively with a co- 
worker. The term could also be more broadly defined as to include managing projects (although 
it was clear that newer Instructional Designers are members of the team rather than managers). 

As one of the telephone participants said “There is not much work that is not team based”, thus 
underscoring the essentialness of team work in the workplace. 

In order to give the reader a balanced perspective with this small sample, it is important to note 
that although teamwork is important, instructional designers also need to be able to work on their 
own. One participant in the focus group said “If you have to be managed you are in trouble”. 
Although it is important to be part of a team, it is also important to be able to work 
independently. In addition, one of the telephone participants indicated that although he knew 
that teamwork was important at other companies, at his present employer teamwork was not as 
important. 

3. Communication/Writing 

Being an effective team player requires good communication. Because many of the participants 
brought this specific issue up we have placed it in it s own category. In addition to general 
interpersonal skills necessary for good teamwork, the idea of probing and eliciting information 
surfaced during our discussions. Instructional designers also need to be able to communicate 
with clients and be able to use the same listening and probing skills. One of the telephone 
participants indicated a more practical aspect of effective understanding when he said “if the 
customer is aware that you are interested in their business they are more likely to get out of the 
box”. On a related note, two of the telephone participants indicated that knowledge of the 
business world was important for instructional designers to have especially because “the 
business world is much different from the academic world”. 

Under the general rubric of communication, writing was a specific topic that stirred great interest 
amongst the participants. We felt that the participants sensed a real gap in their own education as 
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related to writing: “Writing needs to be concise, the university is big on verbosity. (We have to) 
get rid of the crap”. Many of the participants said that knowledge of technical writing was 
important as everything needed to be documented. Also as designers “you have to look at the 
materials” (even though you have technical writers). 

4. Technical Knowledge 

Technical knowledge was a skill that was brought up but wasn’t stressed as being essential. 

This probably has more to do with the fact that technologies are changing quickly in addition to 
the fact that each company realizes that it uses a set of technologies that they can’t expect every 
designer to be familiar with. One participant stated that his company “can train on technology”. 
This doesn’t mean that instructional designers do not need knowledge of technical issues, they 
should have some experience so as to show they have “technical ability”. 

5. Critical Thinking 

Although related to the probing and eliciting information skills in the section on communication 
above, a more general skill of critical thinking and problem solving was indicated to be an 
important skill. Many of the participants indicated that it is important for designers to be able to 
see the big picture when working on projects. One participant stated that “We need to turn out 
people who think like designers” (although we do not remember the locus of the referent to 
‘we’, we believe this is a call to the department). Designers also need to understand how to 
approach a problem and what to concentrate on during this process. Finally, instructional 
designers need to be able to discriminate if something on course and focused on the goals- which 
relates to a more practical understanding of the ISD process. 

6. Affective 

This topic came up mainly in the focus group discussion when the participants were talking 
about the interviewing process. The key ideas were a sense that someone was an ‘achiever” in 
whatever they did and that they had a good attitude both towards flexibility (“there’s no best way 
to do it”) and an attitude towards learning (“A college degree shows a willingness to learn-an. 
attitude to learn”). 

What is the difference between and expert and novice instructional designers? 

The main difference between expert and novice instructional designers came down to 
experience. Expert designers have more experience and thus are less in need of supervision and 
better able to work independently. This is also manifested in the ability to “probe and elicit that 
is not superficial” thus showing a deeper understanding of the process. 

What do designers do at vour company? 

The participants indicted that most of what we had discussed earlier gave a general sense of what 
‘designers do’. In addition, many talked about the importance of interacting with clients. 

Further probing with the telephone participants gave us the sense that it also included a better 
understanding of the business world. That is, the reality of the instructional design world that 
one gets by working in authentic situations. As one of the participants explained “(Instructional 
designers) have to ensure the learning objectives, and clearly achieve the goals of the client and 
program”. Clearly instructional designers have their feet in both worlds. 



What are the biggest weaknesses vou see in recent graduates who you hire to instructional 
designer positions? 

An issue that was raised in response to this question was that of understanding the learning 
process from the perspective of a teacher. One participant said “There are parallels between ILT 
(instructor lead training) and CBT- you have to understand what works”. In addition, it was 
clear that instructional designers are often called upon to do traditional teacher, whether it be 
“training trainers” or “explaining your part of the instruction to other team members”. 

All three phone participants mentioned varying degrees of lack of knowledge about the business 
world as being a serious problem. These problems are manifested in many different ways: 
“professional presence when interfacing with clients”; inflexibility working with business 
clients especially in the design process (“you need to bend to the needs of business”); and 
“understanding the customer’s business needs in order to recommend training” 

What skills will instructional designers need to succeed in the future? 

In a field in which so much emphasis is put on changing technologies, our participants reminded 
us that good teaching is good teaching. Essentially we will “need the same skills as the past”. 
Instructional designers will never replace the teacher. On the other hand, we are in an industry 
that demands technologies. “The web is all the rage” was a comment by one participant, 
explaining that clients often demand a certain technology. The consensus among the participants 
was that there would be a merging of technologies, providing options for the client. 

V. Conclusions 

The key findings we came away from the focus groups are that companies want instructional 
designers who have EXPERIENCE and demonstrated skill in: 

• An intimate theoretical and practical knowledge of the ISD process, including taking 
projects all the way through the process (rather than working on them piecemeal). This 
would imply that the curriculum should be more project-driven that skill drive. 

• Technical writing- many of the participants indicated a lack of technical writing skill in 
their own education. Instructional designers need to be good concise writers. 

• Taking a customer oriented approach, including understanding the business model and 
being able to communicate with customers. 

• Teaching; being able to do instructor lead teaching as well as being persuasive. 

• Situations which elicit critical thinking skills, including following plans and doing the 
job right when faced with opposition 

VI. Future Needs 

Our focus groups contacted people who hire primarily Masters Degreed instructional designers. 
Different perspectives and data could be conducted with people who hire Ph.D. graduates, EDS 
graduates, and people with the undergraduate minor. As was mentioned at the beginning, 
because of the small sample size these results should be interpreted with caution. A more 
extensive study would need to be undertaken in order to provide more valid results. 



VII. Solutions Bank 



A number of the SLC focus group participants were alumni of the USU IT department. During 
the focus group discussion, participants spontaneously offered ideas of how the USU 
Instructional Technology Department could improve the programs it offers. Suggestions could be 
grouped into the following categories: 

• Modify program offerings and requirements 

• Strengthen relationships with industry 

• Give students a more cohesive and complete experience with ISD 

• Encourage student involvement in the professional IT community 

Modify Program Offerings and Requirements 

Alumni participants felt that there were a number of very important areas in which the program 
provided no training, insufficient training, or non-explicit training. 

Technical Writing - Participants commented that instructional designers must know how to write 
well. Despite this, the program does not offer or require writing courses. Focus group 
participants commented that universities often teach students to be verbose, whereas industry 
requires people to write simply, clearly, and succinctly. Focus group participants recommended 
that the program require students to take technical writing courses. 

Problem Solving - The focus group discussion repeatedly brought up the point that problem 
solving skills are important to an instructional designer’s success. Participants suggested that 
perhaps a course could be added to program that explicitly teaches problem solving skills. 

Analysis - Teach students how to ask the right questions. 

Teaching Experience - Require students to teaching classes. 

Strengthen Relationships with Industry 

Focus group representatives felt that since industry is essentially the customer of the program, 
the department should seek a closer relationship with industry. They stated that the companies 
they represent and likely many others would welcome a closer relationship with the department. 
Specific actions to achieve this include: 

Industry Accreditation - Ask industry to participate in the accreditation of the Instructional 
Technology departments. In this way they could help verify that the programs prepare students in 
the areas that industry feels is important. 

Expanded Internships - Have each course in the program involve work with a company. 

Career Training - Near the beginning of the program provide students with exposure to what 
companies look for. It was suggested that if each student could sit in a discussion similar to the 
focus group discussion we held, it would help them see what they should attempt to gain during 
the program. 

Instructional Templates - Have students work on databases of templates for instruction that can 
continually be tested, refined, and deployed. 
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Give Students a More Cohesive and Complete Experience with ISP 

All of the alumni participating in the focus group stated that the program teaches ISD in a 
piecemeal, fragmented fashion. The felt that this type of training does not give students a big- 
picture view of the process. The made the following suggestions for overcoming this: 

Course Sequences - Requiring ISD classes to be taken in the appropriate sequence. 

Continuity -Require students to take projects from start to finish through the ISD process. 

Internships - Facilitate year long (2 semester) projects / internships with companies to coincide 
with classes. 

Student Latitude - Give students more latitude to choose what projects they work on. 

Walk the Talk - Have students analyze the ISD process in the classes they take. Work with 
professors to make sure they are walking the talk and engaging students. 

Encourage Student Involvement in the Professional Community 

Participants felt like the department should do more to encourage students to participate in 
professional activities. Specific suggestions include: 

Journals, List Serves, and Conferences - Plug students into journals and world- wide thinking 
and publishing. Require them to read and participate in list-serves, discussion groups, and 
conferences. 

Professional Organizations - Establish relationships with and promote professional IT 
organizations in addition to AECT (ISPI). 

VIII. Raw Data Packet 



See attached notes 



3154435443 



P.03 



N Qi> 16-93 TUE 13:21 SU-ERIC 



I R 




US, DEPARTMENT OF EDUCATION 
Office of Educational R$te» rch $nd tmprovmont (OERI) 
Educ*tlon*l R0$purc9$ lnform§ (Ion Confer (ERIC) 

REPRODUCTION RELEASE 

(Specific Document) 



I. DOCUMENT IDENTIFICATION: 



Title; 



A lecds o -p^Tsi & ftwcp/dsigi ~7 



JCWWOfQGy (A. tz>h b fBvfr. UsiiVer&ifo ^ 

TS& Pad-'dAK't/Pdqt. r /9) fe&Ei . 4 McPz 

Souroe: " J 1 J 7 Publication Dale. 



Aulhor(s): 



Corporate Sour 



b*^j\ vp 14$ / y^^/ia/^,/ 

REPRODUCTION RELEASE: ^ 



24 /fa n//7<?f 



in order to disseminate as widely as possible timely and significant materials of interest lo the educational commun.ty, documents 
announced in the monthly abstract journal of the ERIC system, Resources >n Education (RIE). are usually made available lo users 
in microfiche, reproduced peper copy, and efecironic/opncal media, and sold througn me ERlC Document Reproduction Service 
(EDRS) or other ERIC vendors. Credit is given lo the sojrce of each document, and. if reproduction release i$ granted, one of 
the following notices Is affixed to the document. 

If permission is granted lo reproduce the identified document, please CHECK ONE o f the tollcwing options and sign me release 
below. 




Sample sticker to be affixed to document Sample slicker lo be affixed lo document 



Check here 

Permitting 

microfiche 

(4* x 6" film;, 
paper copy, 
electronic, 
and optical media 
reproduction 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



- 









to the educational resources 

INFORMATION CENTER (ERlC)." 



"PERMISSION TO REPRODUCE THIS 
MATERIAL IN OTHER THAN PAPER 
COPY HAS BEEN GRANTED BY 












or here 



Permitting 
reproduction 
in Olher than 
paper copy 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERiC)" 



Level t 



Level 2 



Sign Here, Please 



Documents will be processed as indicaied provided reproduction quality permits. It permission to reproduce is granted, but 
neither box is checked, documents will be processed at Level t. 



“1 hereby grant lo the Educational Resources Information Center (ERIC) nonexclusive permission lo reproduce this document as 
indicated above. Reproduction from the ERIC microfiche or elecUonic/oplical media by persons other than ERlC employees and its 
aystem contractors requires permission from the copyright holder. Exception is made for non-profit reproduction t>y libraries and other 
service agencies lo satisfy iniormation needs of educators in response lo discrete inquiries:’ 


S'flf/ai ns y J ($, J— J 


Position’ A 

7r6re*S6 r 


PrinfjfoNarrfe , * 

PL.N . 




Artdt ’ SS htp 7 Vp J2shruct/*e>4.( \ Tcck fok 

ufittk £hiii L UsiUttrsikj 

U. V Jf¥ 3 2-Z-z-g 50 


r*~\Y3S->74^ZtS*- 





OVER 



